Predicting Cognitive Strategies and Eye Movements in Hierarchical Visual Search

This research advances computational cognitive modeling of visual search, and the synergistic relationship between cognitive modeling and eye tracking. The research examines cognitive models of the perceptual, cognitive, and motor processing involved in the visual search of a hierarchical layout. Two types of visual layouts are searched: unlabeled layouts in which words are arranged in groups but with no hierarchical organization, and labeled layouts in which each group is given a heading that guides the search. The two types of layouts motivate fundamentally different search strategies. The models are post hoc explanatory models of the search time data and a priori predictive models of the eye movement data. The models are evaluated based on the eye movement data. The research demonstrates a methodology and provides guidance for predictive cognitive modeling of visual search. The Visual Search Experiment The visual task studied here is finding a known target in a hierarchically-organized visual layout. Layout items are grouped, and sometimes the groups have useful headings. The task is somewhat analogous to looking for a known item on a web page or a product brochure, which is sometimes organized in a useful manner with groups and group headings, and sometimes arranged with no clear and useful organization. The task is specifically designed to reveal the core strategic components involved in a hierarchical search. The task was presented to sixteen experienced computer users. Figure 1 shows a sample layout from the experiment. The layout has six groups of items, and each group is “labeled” with a heading of XnX, where n is a single numerical digit. Figure 1. A “6-group labeled” layout. The precue, in the top left, would have disappeared when the layout appeared. The target is in group F. The gray text did not appear during the experiment. Participants searched eight different screen layouts for a precued target object. Each layout contained one, two, four, or six groups. Layouts were either labeled or unlabeled. In unlabeled layouts, the XnX group labels did not appear. Each unique layout (such as “6-group labeled”) was presented in a separate block of trials. Target and distractor items, group labels, and the target position were randomly selected for each trial. Description of the Models A number of computational cognitive models were built, using the EPIC cognitive architecture (Executive Process-Interactive Control; Kieras & Meyer, 1997). EPIC captures human perceptual, cognitive, and motor processing constraints in a computational framework that is used to build simulations of human information processing and task execution. As is required to use the architecture, we encoded into EPIC the cognitive strategies that guide the visual search. The following two strategies provided a particularly good fit with the observed data. The noisy-systematic search strategy for unlabeled layouts assumes that people attempt to make a “maximally-efficient foveal sweep”, in which the eyes move to capture everything in the high resolution foveal vision, which is roughly 2° of visual angle in diameter, with as few fixations as possible. The mostly-systematic two-tiered search strategy for labeled layouts assumes that people search the group labels until they find the target group, and then confine their search within that group. Predicted and Observed Eye Movements Eye movements were recorded using the LC Technologies Eyegaze System, a 60 Hz eye tracker that tracks eye movements using the pupil-center and cornealreflection. The a priori predicted and the observed eye movements were compared. Figure 2 shows the predicted and observed eye movements from one trial with an unlabeled layout, and from one trial with a labeled layout. The figure gives an idea of the similarities and differences between (a) the predicted and the observed and (b) unlabeled search and labeled search. Table 1 summarizes comparisons between the predicted and observed eye movements. These data, as well as other aspects of this research, are elaborated in Hornof & Halversion (2003). Unlabeled Layout Labeled Layout